DFEE: Interactive DataFlow Execution and Evaluation Kit

DataFlow has been emerging as a new paradigm for building task-oriented chatbots due to its expressive semantic representations of the dialogue tasks. Despite availability large dataset SMCalFlow and simplified syntax, development evaluation DataFlow-based remain challenging system complexity lack downstream toolchains. In this demonstration, we present DFEE, an interactive Execution Evaluation toolkit that supports execution, visualization benchmarking parsers given input backend database. We demonstrate via complex dialog task: event scheduling involves temporal reasoning. It also diagnosing parsing results friendly interface allows developers examine dynamic corresponding execution results. To illustrate how benchmark SoTA models, propose novel covers more sophisticated scenarios metric on task success evaluation. The codes DFEE have released https://github.com/amazonscience/dataflow-evaluation-toolkit.